Multi-Modal Automation for Human Interactive Skill Assessment

ABSTRACT

A method for screening applicants includes remotely accessing a screening entity&#39;s web page by a potential candidate. The candidate is asked, via the pre-screening web page, to input responses to a set of queries tailored to the screening entity and the candidate&#39;s responses to the queries are stored. A phone call is automatically initiated to the candidate in response to at least one of the candidate&#39;s responses. An interactive information gathering process is started with the candidate using both the telephone and the computer. In the process, audio prompts to the candidate are generated via the telephone and the candidate&#39;s verbal responses are recorded via the telephone. A profile of the candidate is stored in a database including the verbal responses from the candidate for later access and analysis by the screening entity. The recorded responses are analyzed by the screening entity to make quality judgments about the candidate.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority, under 35 U.S.C. § 119, of U.S.Provisional Patent Application No. 60/928,895 filed May 11, 2007, theentire disclosure of which is hereby incorporated herein by reference inits entirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

n/a

FIELD OF THE INVENTION

The present invention lies in the field of off-site customer support, inparticular, in the field of identifying qualified human agents forproviding enhanced customer support. The method can be used,particularly, as a tool to assist in separating hirable agents fromunhirable agents.

BACKGROUND OF THE INVENTION

Customer-support centers rely on trained human agents who possess skillsthat are suited for the type of service that is being delivered. Thereare several skill requirements that are common across call centers. Oneexample of such desirable skills is good speech intelligibility, for aspecific language, when speaking over a telephone to a customer. Anotherdesirable skill is the ability to interact with web tools that utilizescreen monitors, keyboards, and other control devices. For customersreceiving assistance over the telephone, an ideal experience can bedelivered by a human agent that speaks the language well, performs theproper tasks, delivers the right information, and sounds delightfulwhile serving the customer in a timely manner.

In the context of identifying qualified human agents, it is desirable tofind efficient methods for measuring one's ability to combine speakingskills and web interaction to produce good customer experience duringtelephone support scenarios. It is also desirable to find efficientmethods for assessing a person's creativity and ability to express thatcreativity verbally. It is particularly desirable to find efficient waysto predict that one will deliver delightful experiences to customersseeking assistance. It is further desirable to determine if an agent hasaptitude in certain areas of specialization.

BRIEF SUMMARY OF THE INVENTION

The invention provides multi-modal automation for human interactiveskill assessment that overcomes the herein-mentioned disadvantages ofthe heretofore-known devices and methods of this general type and that,in the context of identifying qualified human agents, measures one'sability to combine speaking skills and web interaction to produce apositive customer experience during telephone support scenarios,assesses a person's creativity and ability to express that creativityverbally, predicts who is able to deliver delightful experiences tocustomers seeking assistance, and determines if that agent has aptitudein certain areas of specialization.

The present invention does so by helping automate the identification ofhuman agents possessing these qualities.

The present inventive process can automatically pre-screen call centerapplicants based on pre-defined speech tasks. The application ismultimodal and requires simultaneous telephone and Internet web pageaccess by the applicants. There are critical benefits that becomerealized when telephone interaction is coupled with visual informationprovided through web access. In particular, applicants can readweb-based scripted information into the telephone handset. The telephonespeech can be recorded and analyzed in a variety of ways, includingsubjective human assessment and automated assessment provided by aspeech recognizer. Figures, pictures, or any other form of graphics canalso provide the basis for a speech task. For example, a map with ahighlighted route could be displayed on a web page and the speech taskfor the interviewee could include speaking driving directions over thetelephone. Individual driving instructions would need to be accurate(“head east on” instead of “head west on”) and the street names wouldneed to be pronounced correctly.

The present invention is an automated screening application thatidentifies speech clarity, basic thought process, and experience. Theinvention accomplishes tasks normally performed by call centerrecruiters or supervisors of call center personnel and, thereby reducesthe work load and/or personnel for interviewing potentialcandidates—employees who are, typically, highly compensated (at leastwhen compared to the potential candidates).

The present invention has the many valuable characteristics not presentby any previous automated method for interviewing qualified candidates,including, for example:

1. Highly customizable

2. Highly automated

3. Highly efficient

4. Universal access, any language

5. Content flexibility

6. Adjustable acceptance criteria

7. Remote access, any location

The inventive process is highly customizable and can be optimized foralmost any customer service environment. Examples of customer serviceenvironments include computer technical support, concierges, airlinereservations, utilities, telemarketing, car rentals, vacation planning,roadside assistance, and home security. The type of dialogue that isrecorded can be directly correlated to the line of business. In a laterdiscussion, examples will serve to illustrate the various types ofdialogue that are recorded.

The application is highly automated and consumes a minimum amount ofapplicant time and analysis time by human resource personnel. Usingmulti-modal automation, speech tasks, closely tied to audio and visualcues, are recorded and analyzed to evaluate candidates. Although theexamples provided in this document are in English, the application canbe made available in any language and in various combinations oflanguages where such skills are also being evaluated.

Because the inventive process is automated and available electronically,e.g., over the Internet, the program is accessible by applicants at anytime, and from any location, as long as Internet and telephone accessexist. There is no scheduling required and the application can run standalone without human supervision. From the human resources side,applicant data can be reviewed at any time after being stored and can beconfigured to only require appropriate World Wide Web access includingthe ability to listen to recorded audio, e.g., through an audio wavefile player.

With the foregoing and other objects in view, there is provided, inaccordance with the invention, a method for screening applicants by ascreening entity, including the steps of remotely accessing a screeningentity's pre-screening web page over the Internet by a potentialcandidate using an Internet-accessible computer available to thecandidate, requesting the candidate, via the pre-screening web page andthe computer, to input responses to a set of queries tailored to thescreening entity and storing the candidate's responses to the queries,at least during the response requesting step, automatically initiating aphone call to the candidate over a telephone on a telephone network inresponse to at least one of the candidate's responses, commencing aninteractive information gathering process with the candidate using boththe telephone and the computer by generating audio prompts to thecandidate via the telephone and recording the candidate's verbalresponses via the telephone, storing a profile of the candidate in adatabase including the verbal responses from the candidate for lateraccess and analysis by the screening entity, and analyzing the recordedresponses by the screening entity to make quality judgments about thecandidate.

In accordance with another mode of the invention, there are alsoprovided the steps of hosting the web page with external server andaccessing an internal server that hosts an internal web page of thescreening entity.

In accordance with a further mode of the invention, the phone callinitiating step can be carried out by simultaneously initiating thephone call to the candidate or by initiating the phone call to thecandidate after the candidate has completed responding to the queries.

In accordance with an added mode of the invention, the phone call to thecandidate is initiated with a voice server through a private branchexchange over the telephone network.

In accordance with an additional mode of the invention, the candidate'sidentity is confirmed through a set of identification questions andresponses via one of the telephone and the pre-screening web page.

In accordance with yet another mode of the invention, the profilestoring step is carried out by storing the profile of the candidateduring the information gathering process and/or after the informationgathering process is complete.

In accordance with yet a further mode of the invention, the profilestoring step is carried out by storing the profile of the candidatethrough an entity-secure intranet link.

In accordance with yet an added mode of the invention, the phone callinitiating step is carried out by initiating the phone call only withoutbound calls from the screening entity to, thereby, control access tothe information gathering process.

In accordance with yet an additional mode of the invention, theinformation gathering step is carried out by controlling a number andkind of questions presented to the candidate to simulate a realcall-center seat.

In accordance with again another mode of the invention, the storing stepis carried out selectively and/or continually.

In accordance with again a further mode of the invention, the analyzingstep is carries out with automation by automatically determining aconfidence scoring of the candidate with a computerized speechrecognition device programmed to parse a particular response intoindividual words and to either compare the parsed response to a desiredresponses or to transcribe the parsed response for later use and accessby the screening entity.

In accordance with again an added mode of the invention, the automatedconfidence scoring step is carried out by judging at least an accuracyand an intelligibility of the candidate's speech utilizing a targetphrase represented in a speech recognition grammar.

With the objects of the invention in view, there is also provided amethod for screening candidates by a screening entity, includes thesteps of initiating a job application procedure by having an candidateconnect to a secured web page of a screening entity, querying thecandidate to enter candidate-specific personal identification data andsecurely confirming the identification data, and initiating a phone callto the candidate after the identification data is confirmed. While thecandidate is still communicating with the web page and, during the phonecall, at least one of a set of exercises is carried out. For example, avoice quality exercise is initiated over the telephone by asking thecandidate to repeat spoken phrases over the telephone, recording thecandidate's speech for each phrase for subsequent off-line analysis, andautomatically scoring each recorded phrase with a confidence levelcorrelated with how well the recorded phrase matches an expectedpronunciation. Also, a dictation clarity exercise is initiated byinstructing candidate to read a script sent to a particular web pageviewable by the candidate and, when finished, recording speech from thecandidate for subsequent off-line analysis and applying a confidencescoring to the recorded speech. A question-answer exercise is initiatedby transmitting specific multiple-choice questions over the telephone,prompting the candidate to answer each question after it is presented,recording the candidate's speech for each answer, and automaticallyscoring each recorded answer with a confidence level correlated to howwell the recorded phrase matches an expected pronunciation. A serviceknowledge exercise is initiated to determine the candidate's ability tocomprehend an example of good customer service is and whether thecandidate can intelligently describe such an experience by having thecandidate give a speech regarding such an experience, recording thespeech for subsequent off-line analysis, and applying a subjectiveconfidence scoring to the recorded speech. A graphic comprehensionexercise is initiated to screen the candidate's ability to respond to aprovided graphic by displaying a graphic to the candidate, requiring thecandidate to answer questions regarding the graphic, recording thespoken answers over the telephone, and automatically scoring each answerwith a confidence level correlated with how well the recorded phrasematches the expected pronunciation. A speaking satisfaction exercise isinitiated with the candidate to evaluate the kind of experience that apotential customer will have after speaking to the candidate by givingthe candidate a random one of a series of predefined situations in whichcandidate is required to role-play an operator answering in real-time acall from a virtual customer, recording the conversation between thecandidate and the virtual customer for subsequent off-line analysis, andapplying a subjective confidence scoring to the recorded conversation.After at least one of the exercises is complete, the screening entitystores data recorded for later analysis and informs the candidate thatan application process is complete.

In accordance with again an additional mode of the invention, theidentification data is securely confirmed.

In accordance with still another mode of the invention, the secureconfirming step is carried out by one of directly with the candidate andindirectly through a response-requiring email sent to the candidate'semail address.

In accordance with still a further mode of the invention, the phone callinitiating step is carried out with a voice server immediately after theidentification data is confirmed.

In accordance with still an added mode of the invention, the voicequality initiating exercise step is carried out by storing thecandidate's speech for each phrase for off-line analysis.

In accordance with still an additional mode of the invention, thedictation clarity exercise step is carried out by storing speech fromthe candidate for subsequent off-line analysis.

In accordance with another mode of the invention, the confidence scoringstep is carried out at least one of automatically and with humanjudgment.

In accordance with a further mode of the invention, the question-answerexercise step is carried out by storing the candidate's speech for eachquestion for subsequent off-line analysis.

In accordance with an added mode of the invention, the question-answerexercise step is carried out by transmitting the multiple-choicequestions by at least one of a human voice and a machine voice.

In accordance with an additional mode of the invention, the serviceknowledge exercise step is carried out by storing the speech forsubsequent off-line analysis. In accordance with yet another mode of theinvention, the graphic comprehension exercise step is carried out bystoring the spoken answers for subsequent off-line analysis.

In accordance with yet a further mode of the invention, the graphiccomprehension exercise step is carried out by providing a map as thegraphic and screening the candidate's ability to give drivingdirections.

In accordance with yet an added mode of the invention, the speakingsatisfaction exercise step is carried out by storing the conversationfor subsequent off-line analysis.

In accordance with yet an additional mode of the invention, thecandidate is provided with a timeframe for hearing from the screeningentity or given a call number and a date for checking on the candidate'sapplication.

In accordance with a concomitant mode of the invention, the candidate isprovided with a call number and a date for checking on a status of theapplication.

Other features that are considered as characteristic for the inventionare set forth in the appended claims.

Although the invention is illustrated and described herein as embodiedin multi-modal automation for human interactive skill assessment, it is,nevertheless, not intended to be limited to the details shown becausevarious modifications and structural changes may be made therein withoutdeparting from the spirit of the invention and within the scope andrange of equivalents of the claims.

The construction and method of operation of the invention, however,together with additional objects and advantages thereof, will be bestunderstood from the following description of specific embodiments whenread in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Advantages of embodiments of the present invention will be apparent fromthe following detailed description of the preferred embodiments thereof,which description should be considered in conjunction with theaccompanying drawings in which:

FIG. 1 is a block diagram of high-level components of an exemplarysystem architecture for carrying out the method according to theinvention;

FIG. 2 is a process flow diagram of one exemplary process for carryingout the method according to the invention;

FIG. 3 is a diagrammatic representation of an exemplary web interfacefor receiving applicant information in the method according to theinvention;

FIG. 4 is a diagrammatic representation of an exemplary web interfacefor carrying out verbal-applicant-screening exercises in the methodaccording to the invention;

FIG. 5 is a diagrammatic representation of an exemplary web interfacefor carrying out graphic-applicant-screening exercises in the methodaccording to the invention;

FIG. 6 is a list of an exemplary output queue of applicants to bereviewed by screening entities in the method according to the invention;and

FIG. 7 is a diagrammatic representation of an exemplary web interfacefor reviewing an applicant's screening results in the method accordingto the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Aspects of the invention are disclosed in the following description andrelated drawings directed to specific embodiments of the invention.Alternate embodiments may be devised without departing from the spiritor the scope of the invention. Additionally, well-known elements ofexemplary embodiments of the invention will not be described in detailor will be omitted so as not to obscure the relevant details of theinvention.

Before the present invention is disclosed and described, it is to beunderstood that the terminology used herein is for the purpose ofdescribing particular embodiments only and is not intended to belimiting. It must be noted that, as used in the specification and theappended claims, the singular forms “a,” “an,” and “the” include pluralreferences unless the context clearly dictates otherwise.

While the specification concludes with claims defining the features ofthe invention that are regarded as novel, it is believed that theinvention will be better understood from a consideration of thefollowing description in conjunction with the drawing figures, in whichlike reference numerals are carried forward. The figures of the drawingsare not drawn to scale.

Referring now to the figures of the drawings in detail and first,particularly to FIG. 1 thereof, there is shown an illustration ofhigh-level components of an exemplary system architecture for carryingout the present invention. During an applicant screening process, apotential candidate 1, by operating a computer 3, accesses a linkthrough the Internet 11, also known as the “World Wide Web,” to a server4 that hosts an external web page. From the external web page hosted bythe external server 4, an internal server 9 that hosts a web siteinternal to the applicant screening entity 10 is accessed. Thepre-screening web site in the internal server 9 requests the applicant 1to input responses to various queries tailored to the screening entity10. The responses are stored for later use or, as a course of recordkeeping, in a database 8, for example.

Either simultaneously or thereafter, the candidate 1 is called on atelephone 2 through a private branch exchange (“PBX”) 6 over a telephonenetwork 5. A voice server 13 initiates this call 14 to the applicant 1automatically in response to the candidate's response. The candidate'sidentity can be confirmed (or not) through a set of identificationquestions and responses. After confirmation, an interactive processbegins with the potential candidate 1. The process includes both thetelephone 2 and the computer 3 having access to the Internet 11. Thevoice server 13 generates audio prompts to the candidate 1 and recordsthe applicant's verbal responses. After the candidate 1 completes theexercises, a profile is stored in a database 8 for later access andanalyzing by the Human Resource Department of the screening entity 10,for example, via access through an entity-secure intranet link 12. Theanalysis of the recorded responses assists the screening entity 10 tomake quality judgments about the candidate 1.

As with other Human Resource issues, it is desirable to control accessto this screening process. Internet security can be applied toapplicant's access to the external server 4, and an outbound callingstrategy also can control access to the screening process. During theapplicant screening process, control of the number and kind of questionspresented on the applicant's computer 3 and of the phone calls made tothe applicant's phone 2 screens the applicant's 1 individual ability tofollow instructions and to interact in a multi-modal environment, whichsimulates a real call-center seat. Speech is recorded selectivelythroughout the process (or continually) and is stored for subsequentevaluation. It is important, to every extent possible, to automate theevaluation using a confidence scoring produced by a speech recognitiondevice, in other words, with a sufficiently sophisticatedspeech-recognition process, the responses can be parsed into individualwords and compared to desired responses or transcribed for later use andfast access by the entity's evaluation staff. As phrases are pronounced,the voice server 13, which is able to recognize speech, judges at leasttwo characteristics of the applicant's speech: accuracy andintelligibility. This automated confidence scoring quantitativelymeasures pronunciation quality, for example, for a target phrase that isrepresented in a speech recognition grammar.

An exemplary applicant evaluation process is explained below withreference to the process flow chart of FIG. 2 illustrating one exemplaryembodiment of the present invention. The flow starts at step 200 andmoves directly to step 202 where the applicant 1 connects to a securedweb page and initiates the application procedure. As part of theinitiation, applicant 1 can, for example, read an overview, accept termsof use, and select a <continue> option to advance to the next step inthe procedure. In step 204, the applicant enters their personalidentification data, for example, name, address, email address, and/orcurrent telephone number. It is noted that this data can be confirmeddirectly or indirectly through a response-requiring email sent to theapplicant's email address. An exemplary web-provided screen forreceiving this information is illustrated in FIG. 3.

In step 206, the voice server 13 immediately initiates a phone call toapplicant 1, while applicant 1 is still logged on to the webapplication. In step 208, a voice quality exercise is initiated. Theapplicant 1 is asked to repeat spoken phrases over his/her telephone 2.More specifically, specific phrases of a human or machine voice aretransmitted over the phone line to applicant 1 and applicant 1 repeatsthese phrases/words shortly after each phrase is played. For eachphrase, the applicant's speech is recorded and end-pointed for off-lineanalysis. Each recorded phrase is automatically scored with a confidencelevel that is correlated with how well the recorded phrase matches theexpected pronunciation, as represented in a speech recognition grammarwith highly tuned pronunciation lexicons. With such voice recognitiongrammar, strong accents and mispronunciations, for example, will mapinto low-confidence scores. Conversely, clearly spoken words with properinflections and pronunciations will map into high-confidence scores.

In step 210, a dictation clarity exercise is carried out. To start thisexercise, the applicant 1 can be instructed to continue by eitherselecting specific web link buttons (visual) or by pressing keys on thetelephone keypad. To detect clarity of dictation, applicant 1 isrequired to read a script, which is sent to a particular web pageviewable by the applicant 1. The applicant 1 can be given time to readand study the script before speaking the script into the telephone 2. Toallow for this study time (which can be limited if desired by thescreening entity), the applicant 1 will press a key on the telephonekeypad (a web button can also be used with the appropriate architecture)and then dictate the provided script. Upon completion, the applicant 1can be asked to press a telephone key or web button. Speech from theapplicant is stored for subsequent offline processing. Althoughconfidence scoring can be applied, additional human judgment can by usedbecause tone, volume and other acoustical characteristics are moresubjective than objective and, possibly, can be best analyzed by atrained employee. In this way, the employee can score any and allaspects of how well the script was spoken by the applicant 1.

In step 212, a question-answer exercise is initiated. Reference is madeto “Exercise 3” in FIG. 4, which is a sample instruction to an applicant1. Specific multiple-choice questions are transmitted over the telephone2 by a human or machine voice and the applicant 1 is prompted to answereach question after it occurs. For each answer, the applicant's speechis recorded and is end-pointed for off-line analysis. Each recordedphrase is automatically scored with a confidence level that iscorrelated to how well the recorded phrase matches the expectedpronunciation, as represented in a speech recognition grammar withhighly tuned pronunciation lexicons. As set forth above, strong accentsand mispronunciations map into low confidence scores. If an incorrectanswer is spoken, a pre-defined low confidence score will most likely beassigned. Alternatively, if a correct answer is spoken, a pre-definedhigh confidence score will most likely be assigned.

In step 214, a service knowledge exercise is initiated. This speakingexercise is aimed at discovering whether the applicant 1 can understandwhat good customer service is and whether he/she can intelligentlydescribe such an experience. More specifically, as show in “Exercise 4”in FIG. 4, the applicant 1 is asked, for instance, to describe adelightful service that he/she has experienced. The applicant is allowedtime to think of the experience and how he/she would like to describethe experience to the screening entity within a certain time limit (suchas 2 minutes). It should be noted that any questions can be presented tothe applicant 1 for the purpose of screening the applicant's ability tospeak and respond and the present invention is not limited to only thosequestions related to customer-support experiences. The applicant 1,then, speaks over the telephone (or Internet) after pressing a key, forexample, on the telephone keypad (a web button can also be used). Uponcompletion, the applicant 1 indicates that he/she is finished bypressing a telephone key or web button. The speech presented by theapplicant can be stored for subsequent off-line processing. Althoughconfidence scoring can be applied to the spoken words and sentences,here, human judgment can be given greater weight because the content ofthe speech will not be known ahead of time. In particular, humanjudgment can be used to evaluate the applicant's persona and how wellthe applicant's voice will sound to a customer. Additionally, humanjudgment can be used to score the applicant's grammar, intonation, andgeneral talent in the area of servicing customers. Thus, humansubjectivity is well suited for scoring the content and quality of whatwas spoken.

In step 216, a graphic comprehension exercise is initiated. Thisexercise screens the applicant's ability to respond to a providedgraphic. For example, an image, representing the evaluation material, isdisplayed on a web page that is visible to the applicant 1. Instead ofreading a script, repeating a phrase, answering a spoken question, ordescribing an experience, the applicant 1 is required to deduce answersto questions from the provided image and to speak their answers over thetelephone upon being prompted to do so. For each answer, the applicant'sspeech is recorded and is end-pointed for off-line analysis. Eachrecorded phrase is automatically scored with a confidence level that iscorrelated with how well the recorded phrase matches the expectedpronunciation, for instance, as represented in a speech recognitiongrammar with highly tuned pronunciation lexicons. As before, strongaccents and mispronunciations map into low confidence scores. In thisexercise, a word spoken incorrectly significantly reduces the confidencescore, which may be even further reduced when other confidence-loweringfactors are present, such as improper microphone placement, for example.

An exemplary graphic comprehension exercise is illustrated as “Exercise5” in FIG. 5. This example screens an applicant's ability to givedriving directions. The map graphic is displayed to the applicant 1. Theapplicant 1 is asked to give turn-by-turn driving directions and, ifdesired, a list of possible driving instructions. These instructions canbe listed in random order (as shown) or they can be listed in order fromstart to destination (however, this latter approach removes thedeductive reasoning and cartographic analyses that can be performed withthis exercise). In the most difficult case, the applicant 1 will not beprovided with instructions and will be asked to guide a virtual driverfrom the starting point to the destination.

In step 218, a speaking satisfaction exercise is initiated. Thisspeaking exercise is aimed at determining the kind of experience that acustomer will have after speaking to the applicant 1 (such as delight,satisfied, displeasure, horror), the experience type can be referred toas a “pleasantry factor.” The applicant is given one of a series ofrandom situations in which they will be required to role-play anoperator answering a call from a driver (random and/or coordinatedselection is desired where an applicant 2 can enter the applicationprocess more than once and it is insured that a different scenario isrole-played every subsequent time). It is desirable to not give theapplicant 1 time to think and prepare because a “real-time” operatorassistance experience is the desired output. The applicant 1 can beallowed to listen to a driver, for example, one who has just witnessedan accident, who has been involved in an accident, who has locked theirchild in a car, and many other scenarios, and then be asked to counseland assist the driver. The conversation between the applicant 1 and thevirtual driver is stored for subsequent off-line processing. Like step216, only limited automatic confidence scoring can be applied to showuse of grammar and pronunciation, for example. In this case, subjectivehuman judgment is most important to score how well the potentialoperator dealt with the supplied situation.

Any number of other kinds of additional exercises can be performed asdesired in step 220. Once all exercises are complete, the applicant 1 isinformed that the application process is done and the data is ready foranalysis. If desired, the applicant 1 can be given a timeframe forhearing from the screening entity or given a call number and a date forchecking on his/her application. The above exercises are all notnecessary or required. These exercises can occur in any order and in anycombination, some of which can be eliminated if desired.

The process of the present invention now allows HR personnel 10 toreview applicant's stored data 8, step 224, at any time, whether througha web access 9 or after it has been stored internally, in step 222, atthe screening entity's selected data storage location. The process endsat step 226.

Many different screening entities can be allowed to access their own orany other entity's screening data. For example, where an applicant isdetermined to be less suitable for one kind of employment opportunity,that person's performance may be suitable for another opportunity andhaving the data available may be beneficial if different entities agreeto share the screening exercises and recorded results. As such, anadministrative web page 9 can be accessed from an intranet link 12 orfrom any resource connected to the Internet 11, provided that sufficientand/or desired security requirements are met. In order to review therecorded data, measures for playing recorded audio, such as audio wavefiles, to the human resources [or else define it] personnel 10 areneeded.

After an applicant 1 completes the set of exercises, an audio profile(e.g., a web page audio profile) is automatically created, specific tothat applicant 1. For automated and first-in-first-out processing of allapplications received by the screening entity, applicant-specificidentification data can be placed into a queue, as illustrated in FIG.6, for example, for convenient access and processing by human resourcespersonnel, such as over the World Wide Web. To access data regarding aspecific applicant 1, a “score applicant” link can be selected. Phonenumbers in FIG. 6 can be 4-digit internal extensions as well as 10-digitexternal phone numbers.

Upon selecting a specific applicant 1 from the queue, any informationcan be displayed. For example, a task description, correspondingrecorded audio data, and corresponding confidence scoring 13 can bedisplayed for each of the exercises. For each task within an exercise,wave files (for example) are available for listening by the reviewingagent. Displayed with the wave files are associated confidence scoresthat range from 0.9999 to 0.0000, with the higher confidence scoreindicating that the pronunciation is more likely correct than not. Inaddition to automated scoring with confidence measures, the reviewingagent can subjectively score each wave file on a scale of 1 to 100, forexample. Accents, speaking skills, and perceived personality are readilydetected by a reviewing agent trained to screen such candidates based onhow they sound. After listening to each wave file, a subjective score(between 1 and 100) is entered into the applicant's audio-web profilefor future processing. In the example of FIG. 7, subjective scores areshown to be 100, 80, and 80, respectively, for each of three evaluateditems.

For questions that require correct answers (as opposed to repeatingphrases or reading text), a confidence score can serve multiplepurposes, for example, a likelihood of correctness and a likelihood ofcorrect pronunciation. For such questions, high scores are only possiblewhen the correct answer is given and the pronunciation matches therecognizer's expected pronunciation rules, which are represented in avoice-recognition lexicon that can be optimized for specific desiredpronunciations.

For most audio wave files that are recorded by the application,meaningful automated scoring is achieved by applying confidence scoring,which is important to the invention and is described, for example, in“Recognition Confidence Scoring for Use in Speech Understanding Systems”Hazen et al. 2000 (http://citeseer.ist.psu.edu/hazenOOrecognition.html),which is hereby incorporated herein by reference in its entirety. Infact, a completely automated screening process can be used to filter outa high percentage of applicants without human intervention. A standardof acceptance can be adjustable. Performance criteria can be completelyobjective. For example, just by looking at the queue of applicants, theaggregate confidence score (without human intervention) can be displayedand the applicants can be rank-ordered automatically before any humananalysis of the applicant's audio data is performed. Perhaps only thetop 25% of the applicants that complete the screening application willbe considered for further evaluation by human intervention in oneexemplary screening method.

There are several types of applicant-related tasks that can beautomatically scored by applying confidence measures, including, forexample:

1) repeating phrases through prompting.

2) speaking (or reading out loud) displayed text.

3) speaking answers to prompted questions; and

4) speaking answers related to image information.

It is noted that several types of cognitive processing are required tocomplete all of the applicant tasks successfully. These include, but arenot limited to, reading, listening, speaking, knowledge of a language,analyzing images, understanding instructions, being creative, manualdexterity, and possessing relevant knowledge to answer questions. It isthrough the combination of applying various skills that high-confidencescoring is achieved. With appropriate pre-preparation, the processaccording to the present invention is capable of simulating an actualworking environment. In fact, an applicant may be qualified to be avirtual agent (qualified to work remotely) by scoring high enough from“his or her” calling environment, which must include appropriatetelephone and web access.

Various servers 4, 9, 13 are mentioned herein. Mentioning themseparately is not a requirement to being physically separated servers.Accordingly, a single physical server can host the functions describedherein as servers 4, 9, 13.

The foregoing description and accompanying drawings illustrate theprinciples, preferred embodiments and modes of operation of theinvention. However, the invention should not be construed as beinglimited to the particular embodiments discussed above. Additionalvariations of the embodiments discussed above will be appreciated bythose skilled in the art.

Therefore, the above-described embodiments should be regarded asillustrative rather than restrictive. Accordingly, it should beappreciated that variations to those embodiments can be made by thoseskilled in the art without departing from the scope of the invention asdefined by the following claims.

1. A method for screening applicants by a screening entity, whichcomprises: remotely accessing a screening entity's pre-screening webpage over the Internet by a potential candidate using anInternet-accessible computer available to the candidate; requesting thecandidate, via the pre-screening web page and the computer, to inputresponses to a set of queries tailored to the screening entity andstoring the candidate's responses to the queries; at least during theresponse requesting step, automatically initiating a phone call to thecandidate over a telephone on a telephone network in response to atleast one of the candidate's responses; commencing an interactiveinformation gathering process with the candidate using both thetelephone and the computer by: generating audio prompts to the candidatevia the telephone and recording the candidate's verbal responses via thetelephone; storing a profile of the candidate in a database includingthe verbal responses from the candidate for later access and analysis bythe screening entity; and analyzing the recorded responses by thescreening entity to make quality judgments about the candidate.
 2. Themethod according to claim 1, which further comprises: hosting the webpage with external server; and accessing an internal server that hostsan internal web page of the screening entity.
 3. The method according toclaim 1, which further comprises one of: carrying out the phone callinitiating step by simultaneously initiating the phone call to thecandidate; and carrying out the phone call initiating step by initiatingthe phone call to the candidate after the candidate has completedresponding to the queries.
 4. The method according to claim 1, whichfurther comprises initiating the phone call to the candidate with avoice server through a private branch exchange over the telephonenetwork.
 5. The method according to claim 1, which further comprisesconfirming the candidate's identity through a set of identificationquestions and responses via one of the telephone and the pre-screeningweb page.
 6. The method according to claim 1, which further comprisescarrying out the profile storing step by storing the profile of thecandidate one of: during the information gathering process; and afterthe information gathering process is complete.
 7. The method accordingto claim 1, which further comprises carrying out the profile storingstep by storing the profile of the candidate through an entity-secureintranet link.
 8. The method according to claim 1, which furthercomprises carrying out the phone call initiating step by initiating thephone call only with outbound calls from the screening entity to,thereby, control access to the information gathering process.
 9. Themethod according to claim 1, which further comprises carrying out theinformation gathering step by controlling a number and kind of questionspresented to the candidate to simulate a real call-center seat.
 10. Themethod according to claim 1, which further comprises carrying out thestoring step one of selectively and continually.
 11. The methodaccording to claim 1, which further comprises carrying out the analyzingstep with automation by automatically determining a confidence scoringof the candidate with a computerized speech recognition deviceprogrammed to parse a particular response into individual words and toeither compare the parsed response to a desired responses or totranscribe the parsed response for later use and access by the screeningentity.
 12. The method according to claim 11, which further comprisescarrying out the automated confidence scoring step by judging at leastan accuracy and an intelligibility of the candidate's speech utilizing atarget phrase represented in a speech recognition grammar.
 13. A methodfor screening candidates by a screening entity, which comprises:initiating a job application procedure by having an candidate connect toa secured web page of a screening entity; querying the candidate toenter candidate-specific personal identification data and securelyconfirming the identification data; initiating a phone call to candidateafter the identification data is confirmed and while the candidate isstill communicating with the web page and, during the phone call,carrying out at least one of the following exercise steps: initiating avoice quality exercise over the telephone by asking the candidate torepeat spoken phrases over the telephone, recording the candidate'sspeech for each phrase for subsequent off-line analysis, andautomatically scoring each recorded phrase with a confidence levelcorrelated with how well the recorded phrase matches an expectedpronunciation; initiating a dictation clarity exercise by instructingcandidate to read a script sent to a particular web page viewable by thecandidate and, when finished, recording speech from the candidate forsubsequent off-line analysis and applying a confidence scoring to therecorded speech; initiating a question-answer exercise by transmittingspecific multiple-choice questions over the telephone, prompting thecandidate to answer each question after it is presented, recording thecandidate's speech for each answer, and automatically scoring eachrecorded answer with a confidence level correlated to how well therecorded phrase matches an expected pronunciation; initiating a serviceknowledge exercise to determine the candidate's ability to comprehend anexample of good customer service is and whether the candidate canintelligently describe such an experience by having the candidate give aspeech regarding such an experience, recording the speech for subsequentoff-line analysis, and applying a subjective confidence scoring to therecorded speech; initiating a graphic comprehension exercise to screenthe candidate's ability to respond to a provided graphic by displaying agraphic to the candidate, requiring the candidate to answer questionsregarding the graphic, recording the spoken answers over the telephone,and automatically scoring each answer with a confidence level correlatedwith how well the recorded phrase matches the expected pronunciation;and initiating a speaking satisfaction exercise with the candidate toevaluate the kind of experience that a potential customer will haveafter speaking to the candidate by giving the candidate a random one ofa series of predefined situations in which candidate is required torole-play an operator answering in real-time a call from a virtualcustomer, recording the conversation between the candidate and thevirtual customer for subsequent off-line analysis, and applying asubjective confidence scoring to the recorded conversation; and after agiven number of exercises are complete, storing data recorded for lateranalysis by the screening entity and informing the candidate that anapplication process is complete.
 14. The method according to claim 13,which further comprises securely confirming the identification data. 15.The method according to claim 13, which further comprises carrying outthe secure confirming step by one of directly with the candidate andindirectly through a response-requiring email sent to the candidate'semail address.
 16. The method according to claim 13, which furthercomprises carrying out the phone call initiating step with a voiceserver immediately after the identification data is confirmed.
 17. Themethod according to claim 13, which further comprises carrying out thevoice quality initiating exercise step by storing the candidate's speechfor each phrase for off-line analysis.
 18. The method according to claim13, which further comprises carrying out the dictation clarity exercisestep by storing speech from the candidate for subsequent off-lineanalysis.
 19. The method according to claim 13, which further comprisescarrying out the confidence scoring step at least one of automaticallyand with human judgment.
 20. The method according to claim 13, whichfurther comprises carrying out the question-answer exercise step bystoring the candidate's speech for each question for subsequent off-lineanalysis.
 21. The method according to claim 13, which further comprisescarrying out the question-answer exercise step by transmitting themultiple-choice questions by at least one of a human voice and a machinevoice.
 22. The method according to claim 13, which further comprisescarrying out the service knowledge exercise step by storing the speechfor subsequent off-line analysis.
 23. The method according to claim 13,which further comprises carrying out the graphic comprehension exercisestep by storing the spoken answers for subsequent off-line analysis. 24.The method according to claim 13, which further comprises carrying outthe graphic comprehension exercise step by providing a map as thegraphic and screening the candidate's ability to give drivingdirections.
 25. The method according to claim 13, which furthercomprises carrying out the speaking satisfaction exercise step bystoring the conversation for subsequent off-line analysis.
 26. Themethod according to claim 13, which further comprises providing thecandidate with a timeframe for hearing from the screening entity orgiven a call number and a date for checking on the candidate'sapplication.
 27. The method according to claim 13, which furthercomprises providing the candidate with a call number and a date forchecking on a status of the application.